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Executive summary 

Aims, method and sample 

This research, funded and supported by the Australian Flexible Learning Framework 
(Framework) 1 , examines what practitioners and auditors judge to be the key issues in 
e-assessment, and most significantly, potential practical actions that will promote 
better understanding and improved practices. In this research, e-assessment covers 
the wide range of activities where digital technologies are used in assessment, 
including the designing and delivery of assessments, and in marking. 

In total, 48 individuals provided input into this study, either by completing interviews, 
teleconferences or small group discussions. Interviews were about one hour in 
length, and a semi-structured approach was adopted so that additional questions and 
probes were also used. Participants were located through advice from a Reference 
Group, and from other participants, while we attempted to sample practitioners and 
auditors from across Australia. Participants were expert practitioners, expert auditors 
or others who had in-depth experiences with e-learning and audit in the vocational 
education and training (VET) sector. 

Three case studies 

To support the findings from the interviews, three case studies of e-learning 
assessment practices in the VET system were also developed. The three cases are 
available on the Framework website (see http://flexiblelearninq.net.au/research) . The 
cases are titled: 

• Vocational training for veterinary nurses - the case study describes the 
continuing journey at Brisbane North Institute of TAFE to increase its online 
learning focus within Veterinary Nursing training. The Veterinary Nursing 
project team have experienced a paradigm shift around how to best use new 
technologies to deliver training and associated assessment. Significantly, this 
change is now being linked to improvements in the overall learning 
experience of learners in Veterinary Nursing qualifications 

• The el_F program - this case study describes how TAFE NSW - Western 
Sydney Institute have designed this program to allow teachers to experience 
a range of activities which include facilitation strategies and the use of 
emerging educational technologies. Completing these activities provides 
teachers with the skills to develop and facilitate their own learning activities, 
units and courses using appropriate technologies and an e-learning facilitated 
approach 

• E-examinations - at the University of Tasmania, its Faculty of Education has 
developed an open source operating system on a live CD, and subsequently 
converted to USB drives, that provides a holistic examination environment 
which learners use to practise personal skills at home without copyright 
infringement. The UTAS e-examination system has now been used for over 
three years, with increasing sophistication over time. 



1 The Framework is the national training system's e-learning strategy: 
http://flexiblelearning.net.au 
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Formative and summative assessment 

Practitioners and auditors believed that e-assessment was finding its place in the 
VET sector mostly as a form of diagnostic and formative assessment. The primary 
use of e-assessment for summative assessment was rare. There was little support 
from practitioners or auditors for the use of e-assessment as the primary form of 
evidence to support the signing-off of a formal qualification or certification of a skill. 
However, practitioners and auditors envisaged considerable growth in the future 
around the incorporation of various forms of e-examinations, short e-quizzes during 
training, and multiple choice questionnaires to test knowledge. In contrast, 
interviewees did not expect to see any major growth around the application of 
e-assessment around complex high stakes skills. 

While it was noted that Toolboxes were not designed for assessment tasks, they 
were being used for formative assessment. Practitioners were slowly becoming more 
aware of the potential for Toolboxes around e-assessment, especially as Toolboxes 
do involve projects that can be completed online. But many of those interviewed 
believed that this potential needed to be more fully promoted to VET practitioners. 

Maintaining quality standards 

According to auditors, the majority of audits confirmed that quality assessment 
practices were occurring. They provided examples where the standards were applied 
as rigorously or even more rigorously to e-learning and e-assessment, as for 
traditional training and assessment. According to our expert practitioners, some 
issues around the quality of e-assessment can be linked to assumptions by some 
teachers that everything to do with e-assessments is located online. As a result, 
teachers do not prepare a study guide, a unit assessment plan or the assessment 
marking criteria. Also practitioners and auditors reported a set and forget' attitude 
around some forms of e-assessment. 

Maintaining confidence in assessment 

Respondents were positive about the operation of the Australian Quality Training 
Framework (AQTF) 2007. The outcomes-focused approach being adopted in the 
AQTF was well understood, while practitioners believed that the principles of 
continuous improvement were applied by internal and external auditors. However, 
practitioners and auditors were still coming to grips with the newer and more varied 
forms of assessment. Today there is a larger range of assessment strategies, 
processes and instruments than in the past and more complexity in the evidence 
being captured by these new forms of assessment. Even highly skilled practitioners 
were having their professional judgements around e-assessment challenged in 
audits. 

At present, tensions do exist between practitioners and auditors around various 
forms of e-assessment. These tensions are around maintaining the reliability and 
validity of assessment, while also achieving flexibility and cost efficiencies. A central 
factor is maintaining confidence in the assessment system. Some practitioners and 
their institutions are confused around the expectations and decisions of auditors 
around e-assessment. However, auditors believed that they were very consistent in 
the messages they gave in their meetings with practitioners. 

Audit experiences 

Auditors reported that the forms of e-assessment currently being audited were not 
diverse. The dominant e-assessment methods being audited included online tests to 
assess knowledge, and the use of blogs and wikis to promote reflection and self- 
assessment. The use of e-portfolios was on the increase. Auditors responded that 
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their position in any audit is not to hold any preconceptions about the form that 
evidence may take. Regardless of the evidence gathering methods used, auditors 
reported that they want to see evidence that the training and assessment is meeting 
clients' needs, the principles of assessment and rules of evidence, and the 
assessment is aligned to AQF level. They want to be able to confirm dimensions of 
competency, employability skills and transferability. This message was a very 
consistent one in the auditor interviews. 

Some auditors highlighted how the demands upon their time possibly encouraged 
decisions around the validity of assessment and e-assessment to fall on the side of 
caution. Many auditors believed that they were 'naturally suspicious people', and 
what they had seen around the poor use of e-assessment around authenticity 
supported their inclination to be cynical and suspicious. Others noted that despite 
considerable opportunities for professional development, many auditors were 'still 
wedded to tradition and paper evidence more than we should be'. Current auditor 
training did not provide wide exposure to e-learning or e-assessment systems. 

Issues around validity 

Validity was a major concern for auditors. They were seeing poor assessment 
practices at work. E-assessments were not validly assessing the skills being tested 
and the assessment did not always address the intended learning outcomes. Some 
e-assessment had been introduced without establishing learner needs and 
completing training needs analysis. Auditors emphasised the need for more use of 
multiple assessments, independent validation from subject matter experts, and 
possibly mandatory interviews at the need of the learning and prior to judgements 
about competency. Auditors also wanted to see better practices around evidence 
gathering through multiple sources such as observations, testimonials and work- 
based assignments to support the validity of the online assessments. 

Issues around authentication 

Audits revealed issues around delivery that included poor security, unreliable 
infrastructure that impacted upon the user in completing online assessment, and 
accessibility and ease of use. On the other hand, practitioners mentioned more trials 
around technological solutions to deal with many of the challenges to authenticity. It 
was widely accepted by those interviewed that the right technology can enhance the 
security of e-assessments. However, the position shared by practitioners and 
auditors was that high quality and supportive relationships between teachers, their 
online learners, and with employers were also critical to managing issues around 
authenticity. 

The online quiz 

The predominant example of e-assessment is the online quiz. Practitioner users of 
quizzes cited advantages around the preparation and marking; assessing learner 
knowledge quickly; and providing more regular feedback. However, disadvantages 
included poorly constructed online tests with limited validity and reliability. It was 
reported by practitioners and auditors that some teachers have little or no 
understanding of how to design valid and fair online quizzes, especially 
underestimating the skills and effort needed to write e-assessments. At worst online 
tests and short quizzes were described, by a number of auditors and practitioners, as 
'cheap options'; and 'ill-thought out efforts by the provider to make money quickly or 
to save money'. 
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Assessment support materials 

Respondents recognised gaps in assessor support materials around competency- 
based assessment in these new online learning environments. A major concern was 
the plethora of poor quality e-assessments tools being used online, on intranets and 
in blended learning. Both practitioners and auditors noted that due to inexperience 
some teachers are failing to identify these poor quality tools. More practitioners were 
becoming aware of the potential for Flexible Learning Toolboxes (Toolboxes 2 ) around 
e-assessment, especially as Toolboxes do involve projects that can be completed 
online. 

The use of e-portf olios 

Practitioners and auditors reported growth in the use of e-portfolios in training 
organisations. Practitioners noted that e-portfolios can streamline evidence and 
validation, while the authenticity of evidence can be verified by appropriate learning 
management systems/strategies. Some auditors, however, had concerns about 
e-portfolios as audits had raised serious questions around the authenticity of 
evidence gathered by learners and the validity of the assessment tasks given for 
units of competency. Practitioners also reported a reluctance to adopt e-portfolios 
due to the lack of guidelines and good examples around structuring suitable pieces of 
assessment. 

Auditor consistency and training 

Practitioners appreciated the cross-jurisdictional experience that some auditors now 
brought to the audit. Such experiences helped both practitioners and auditors 
become more confident in recognising good practice around assessment and 
e-assessment. However, some training providers delivered training across multiple 
jurisdictions had experienced different audit outcomes for the National Audit and 
Registration Agency (NARA 3 ) and other audits. 

A number of ideas were put forward by our expert auditors about how to promote 
better levels of consistency among auditors. One strategy is to allocate the task of 
training auditors to a third party with no direct interest in the outcome. To improve 
audits around new issues like e-assessment, there was some discussion about the 
need by auditors or auditing organisations to engage registered training 
organisations (RTOs) at an earlier stage than at present. Auditors were very up front 
about the need to continue to broaden their understanding of e-learning and 
e-assessment. 

More well trained and informed practitioners 

Many practitioners and auditors proposed that the assessment knowledge and skills 
of VET practitioners must be enhanced. The current VET practitioner needs to be 
more able to: 



• 



possess the technical skills to devise or create valid, reliable and authentic 
assessment and e-assessment 

demonstrate knowledge and understanding of assessment best practice 

understand and address issues relating to the use of e-assessment 



2 Toolboxes are high quality, cost effective interactive e-learning and assessment resources: 
http://flexiblearning.net.au/toolboxes 

3 NARA provides audit and registration services for RTOs that operate in more than one 
Australian state or territory: http://www.nara.tvetaustralia.com.au/ 
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• access e-tools or resources that are easy to use and cost effective 

• access high quality continuing professional development courses that allows 
them to become familiar with the principles and best practice of 
e-assessment. 

The need for worked examples of e-assessment 

Practitioners and auditors agreed that a wide range of the traditional assessment 
tools can work effectively in the online environment. However, as with traditional 
approaches, assessment strategies need to be selected to fit the learner's abilities 
and experience, and to match the outcomes that need to be achieved. In addition, 
practitioners want access to better examples that illustrate the use of quality 
e-assessment. They require examples from both practitioners and auditors around 
e-assessment that meets the requirements of the relevant training package or 
course. 

The need to formulate e-assessment guidelines 

As noted by numerous interviewees, the VET sector has not put enough thought into 
what the guidelines are for online delivery and assessment. Currently those working 
in the sector are confused. They are trying to make their processes appear to fit 
guidelines that are grounded in paper-based approaches to assessment. Most 
respondents talked of the need for a separate handbook or resource that was a 
collection of examples that address all assessment, and include examples in both 
traditional and e-assessment modes. 

Next steps 

There needs to be continued confidence around traditional assessment and in the 
new forms of assessment that will emerge. Significantly, those interviewed agreed 
that the time is right to pull together a shared understanding of the key issues. 
Importantly, this report also puts forward ideas that might promote a greater 
understanding by both auditors and practitioners of more specific actions around 
building confidence in e-assessment. 

Following on from the findings from our interviews, we propose further discussion of 
the following ideas: 

1 . Preparation of guidelines for e-assessment in the VET sector titled 
E-assessment: Guide to effective practice in the VET sector (Guide). 

2. Collation of examples of good practice around the use of a wide variety of 
e-assessments. 

3. The delivery of workshops that promote a broader discussion between 
practitioners, auditors and others of the key issues raised in this report. 

4. Preparation of support materials around the design and delivery of online 
quizzes. 

5. The development of units of competency in e-assessment that become part of 
any planned revisions to the current training and assessment (TAA). 

6. An independent organisation for e-assessment that provides support for 
professionals in the field, collecting best practice guidelines and 
communicating the positive aspects of e-assessment. 

7. Towards assisting in the preparation of the Guide and best practice 
examples, to complete a state-of-the-art review of techniques in 
e-assessment which should be considered for application in the VET sector. 
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Introduction and aims 

Assessment that is judged to be valid, reliable, fair and flexible is a key pillar of the 
Australian vocational education and training (VET) system. Within this system, the 
Australian Quality Training Framework (AQTF) 2007 is the set of standards which 
assures nationally consistent, high-quality training and assessment services for the 
clients of Australia's VET system. The compliance requirements under the AQTF 
emphasise the importance of consistency in a national VET assessment system, 
while the principle of mutual recognition requires that all registered training 
organisations (RTOs) recognise the qualifications issued by other training providers. 

This project aimed to further our understanding of what practitioners and auditors 
perceive to be the key issues in e-assessment, and potential strategies that will 
maintain and promote greater understanding and improved practices. More forms of 
assessment are using new technologies, and as with any assessment in the VET 
sector, they require auditing according to the national standards of AQTF 2007. Our 
particular focus is upon the views of practitioners and auditors about what can be 
done to promote greater confidence in the quality and consistency of the assessment 
processes and judgements in the VET sector. The research therefore puts forward 
practical recommendations that will be presented and discussed further at various 
forums throughout 2010, including planned workshops with auditors. 

Up front it is worth noting that in a project like this, opinions will differ about 
assessment due to the reality that assessment has to complete a 'double duty' 4 . It 
needs to be rigorous, but not exclusive. It must be authentic, yet reliable. 
Assessment needs to be exacting, while also being fair and equitable. Assessment 
needs to meet long-established standards, but it must reflect and adapt to 
contemporary needs. Behind some of these tensions is the premise that assessment 
at one and the same time needs to accommodate not only the expectations of 
teachers, their students and the institution in which they are engaged, but also the 
expectations of governments, employers, professional and accrediting organisations 
and the public at large. 

The significance of the current project 

The current project is very timely and it is significant for several reasons. For the 
Australian VET sector, there is a critical need to maintain high quality training and 
assessment services for the clients of Australia's VET system. We need to keep 
getting our assessment 'right'. Getting assessment right is essential to the continued 
well-being of our VET learners and institutions. Getting assessment right is also 
critical to keeping Australia internationally competitive through having a skilled and 
flexible workforce. 

Effective assessment is one of the best ways to identify the support needed by 
learners so that they continue to develop the required levels of knowledge and skills. 
Successful learning is linked not only to good quality and timely feedback, but also to 
challenging and stimulating forms of assessment that allow learners to demonstrate 
their understanding and skills. Therefore, there needs to be continued confidence in 
the assessment processes for all stakeholders in the Australian system. 

This project also informs the ongoing debates about assessment in the VET sector. 
This debate has already examined inconsistencies in assessment practices around 



4 Boud, D. (2007) "Great designs: what should assessment do?" REAP International Online 
Conference on Assessment Design for Learner Responsibility, 29-31 May. Available at 
http://ewds.strath.ac.uk/REAP07 ; Marton, F., Hounsell, D. & Entwistle, N. (1996). The 
Experience of Learning: Implications for Teaching and Studying in Higher Education. Second 
Edition. Edinburgh: Scottish Academic Press. 
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the requirements of training packages, and questions around how rigorous training 
providers are in terms of quality assurance of assessment. There are concerns about 
the initial and ongoing training for assessors and auditors in both the traditional and 
more contemporary forms of assessment 5 . In addition, there are some suggestions 
that AQTF auditing is a major barrier to the use and growth of e-learning 6 . There are 
suggestions of growing tensions and a perceived divide between practitioners, 
assessors and auditors as AQTF auditors are seen to be suspicious of evidence 
provided through e-learning assessment. 

The fifth annual E-learning Benchmarking Survey of the Australian Flexible Learning 
Framework (Framework) 7 reveals the continued growth in e-learning and 
e-assessment practices 8 . The 2009 survey results show that 46% of RTOs were 
using e-learning for assessment. Among teachers and trainers delivering units using 
e-learning, 62% nominated its use in online assessment activities. Moreover, this 
figure represents a considerable increase on responses to this question in the 2007 
and 2008 E-learning Benchmarking Surveys. To more fully debate the implications of 
this growth in e-learning and associated e-assessment, we need to be better 
informed about the opinions of the teachers, learners and auditors who are major 
agents in supporting or hindering the adoption of these innovations in learning and 
assessment. 

This report is not alone in being focused upon the need for ensuring the continued 
confidence in our assessment practices in the VET sector. The National Quality 
Council (NQC 9 ) continues to oversee quality assurance and national consistency in 
the application of the AQTF 2007 standards for the audit and registration of training 
providers. Through the National Consistency Action Group (NCAG), the NQC is 
focused upon achieving greater national consistency and in developing the 
capabilities of auditors and RTO staff with responsibility for assessment and 
implementing AQTF standards. In addition, the NQC through the Quality Assessment 
Action Group (QAAG) has undertaken a range of research directed at improving the 
quality of assessment including the validation and moderation of all aspects of 
assessment 10 . 

Among these publications is practical resource material for assessors and assessor 
trainers seeking technical guidance on how to develop or review assessment tools. 
The guides being provided are not intended to be mandatory, exhaustive or 
definitive, but to be aspirational and educative in nature. 

Finally, the Framework, as confirmed in its 2008-2011 Framework Strategy 11 , is 
focused upon how better integration and embedding of e-learning in the practices of 
training providers and businesses can be achieved. 



5 Clayton, B., Roy, S., Booth, R., & House, R. (2004) Maximizing confidence in assessment 
decision-making; Current approaches and future strategies for quality assurance. Adelaide: 
ANTA. 

6 Figgis, J., & Guthrie, H. (2009). Moving forward with research. Australian Flexible Learning 
Framework. Canberra: DEEWR. 

7 The Framework is the national training system's e-learning strategy: 
http://flexiblelearninq.net.au 

8 Australian Flexible Learning Framework (2009) E-learning benchmarking survey. Canberra: 
DEEWR http://e-learningindicators.flexiblelearninq.net.au/survey results all. htm 

9 http://www.nqc.tvetaustralia.com.au/ 

10 Publications are located at: http://www.nqc.tvetaustralia.com.au/nqc publications 

11 http://www.flaq.edu.au/content/2008-201 1 -framework-strategy 
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AQTF 2007 and the audit process 

A cornerstone of quality assessment principles and practice is the AQTF2007 
Essential Standards for Registration. RTOs are required to ensure that assessment 
meets the requirements of the relevant training package or accredited course. 
Assessment must be conducted by qualified assessors in accordance with the 
principles of assessment and the rules of evidence. The need for continued quality 
judgements by practitioners, assessors and auditors underpins the operation of the 
AQTF 2007. Processes need to be established and maintained that assist AQTF 
auditors in making consistent decisions around the assessment matters raised in 
auditing RTOs. Given this, an interesting issue explored in this report is whether 
auditors are perceived by practitioners to be making consistent decisions around 
newer forms of assessment, such as e-assessment. 

The AQTF auditing principles specify that audits are systematic, outcomes-focused, 
evidence-based, flexible, focused on continuous improvement, fair, open and 
transparent. A primary role for the audit team is to confirm from evidence provided by 
the RTO that the outcomes from the RTO's processes meet the requirements of the 
AQTF 2007 Essential Standards for Registration. At the core of the Standards is 
evidence, as well as evidence that the RTO has an approach around continuous 
improvement in the management of its operations. Auditors' judgements are based 
on the responses by practitioners to several key questions, including: Is the evidence 
valid? Is it clearly related to the standard that is being audited? Has the RTO 
provided sufficient evidence for a judgement about compliance to be made? Is the 
evidence presented by the RTO authentic? Does it require further authentication? 

Finally, it is important to note that the current research supports significant efforts 
over the last decade to strengthen our VET assessment system. These initiatives 
include the introduction of improved quality assurance around assessment through 
the implementation of the AQTF; the enhanced training of assessors; and the 
establishment of well-defined competency standards in most industry sectors 12 . It is 
important that this strengthening continues through generating ongoing constructive 
debate around the continued evolution of the AQTF. 



12 Learning Australia (2009) Revised final report. Quality of assessment practices - Stage 1. 
Canberra: DEEWR. 
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Overview: E-assessment in Australia and 
elsewhere 

Defining e-assessment 

Definitions of e-assessment typically use the concept as a generic term that covers 
all uses of computers in assessment 13 . E-assessment is a broadly-based concept 
that covers a wide range of activities where digital technologies are used in 
assessment, including the designing and delivery of assessments, and in marking. It 
includes the processes of reporting, storing and transferring of data associated with 
public and internal assessments. It is also defined around the end-to-end electronic 
assessment processes where ICT is used for the presentation of assessment activity, 
and the recording of responses. This includes the end-to-end assessment process 
from the perspective of learners, teachers, providers, awarding bodies and regulators 
and the general public. 

The validity of using e-assessment for one or more of the stages of assessment is 
becoming more hotly debated. These stages include diagnostic assessment of a 
learner's knowledge and skills at the outset of a course; to formative assessment that 
provides developmental feedback to a learner on their current understanding and 
skills; and the summative or final assessment of a learner's achievements around a 
set of knowledge or skills. The 'stakes' around the outcomes of assessment also 
differ 14 . A low-stakes assessment is usually formative with the results being about 
feedback that supports and motives the learner. A high-stakes assessment, however, 
is one in which the outcomes are of high importance to both the institution and the 
candidate, impacting upon their progress or their gaining of a qualification. 

Currently, in most countries e-assessment is more frequently found in low- to 
medium-stakes rather than in high-profile, high-stakes contexts. However, this 
situation is changing. Major awarding and regulatory bodies in the United Kingdom 
and Europe in particular are exploring the wider use of e-assessment. These broader 
applications of e-assessment are across more stages and in higher stakes contexts. 

Overseas experiences with e-assessment 

Australia is not alone in grappling with possible standards and guidelines around the 
use of-e-assessment. Large overseas companies have put e-assessment on their 
agenda. For example, in 2009 a Cisco, Intel and Microsoft consortium launched a 
collaborative project around a shared global priority to transform educational 
assessment (Transforming Education: Assessing and Teaching 21st Century Skill). 
Their goal is to mobilise educational, political and business communities around the 
needs and opportunities for transforming educational assessment and instructional 
practices. Their working proposition is that most educational systems operate much 
as they did in the beginning of the 20th century, dragging well behind the 
opportunities provided by new technologies. 



13 Joint Information Systems Committee (2007) Effective practice with e-assessment: 
Overview of technologies, policies and practice in further and higher education. London: 
HCERC; http://www.iisc.ac.uk/media/documents/themes/elearninq/effpraceassess.pdf ; 
Scottish Qualifications Authority (2007) Regulatory guidelines for e-assessment. London: 
Qualifications and Curriculum Authority. 

14 Ibid. 
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In the United Kingdom, there is a growing consensus that e-assessment promises 
many potential benefits ranging from lowered costs, higher productivity and faster 
feedback, through to assessments which are more accurate, more detailed and more 
robust under critical scrutiny and audit. However, there is equally wide concern that 
such benefits are relatively slow to emerge. Progress is being hampered by 
examples of poor quality assessment which fail to reliably provide acceptably valid 
measures of student achievement and capability, particularly at the higher levels of 
Bloom's taxonomy 15 . 

In preparing for the increased use of e-assessment in future decades, the regulatory 
authorities in England, Wales and Northern Ireland are producing regulatory 
principles and guidance for e-assessment 16 . These principles are being supported by 
the development of accredited qualifications in e-learning and e-assessment. 
Supporting the development of these standards and examples of good practice and 
the research reports by the Joint Information Systems Committee (JISC). Its role is to 
promote in colleges and universities in the United Kingdom the innovative use of 
digital technologies towards maintaining UK's position as a global leader in 
education 17 . 

Various reports from the JISC reveal that 18 : 

• the current use of e-assessments is uneven in these countries 

• e-assessment is highly effective where it is used well 

• there are concerns around the appropriate form of regulations that might guide 
new practices and procedures around e-assessment, authentication and the 
training and professional development needs of staff involved in e-assessment 

• more champions are required within institutions to help users deal with the 
technological challenges in particular 

• more widespread use of e-assessment occurs where the preparatory work is 
completed by a third party or other technically expert staff. 

The JISC has also produced the E-assessment: Guide to effective practice to assist 
the various qualifications regulators in the United Kingdom 19 . The guide is linked to a 
new Level 3 award in delivering e-testing. The units that make up this award form 
part of the new Teacher Qualifications Framework being developed by Lifelong 
Learning UK (LLUK), and are integrated directly with the key e-testing roles that the 
guide sets out. The Guide is also linked to the web site provided by Ofqual (the new 
regulator of qualifications, exams and tests in England) and linked to an 
e-assessment toolkit 20 . 



15 Bloom, B.S., Englehart, M.D., Furst, E.J., Hill, W.H., and Krathwohl, D.R., (1956) Taxonomy 
of Educational Objectives: Handbook 1: Cognitive Domain, Longman, New York. 

16 Joint Information Systems Committee (2007) Effective practice with e-assessment: 
Overview of technologies, policies and practice in further and higher education. London: 
HCERC: http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf 

17 See http://www.jisc.ac.uk 

18 Ripley, M. Harding, R., Redif, H., Ridgway, J., & Tafler, J. (2009) Review of advanced e- 
assessment techniques. Final report. JISC. 

19 Qualifications and Curriculum Authority (2007) E-assessment: Guide to effective practice. 
London: QCA. http://ofqual.staging.designbyfront.com/images/uploads/e-assessment - 

guide to effective practice full version.pdf 

20 http://toolkit.efutures.org 
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This guide promotes the use of e-assessment in an advisory rather than a regulatory 
capacity. It offers practical information and advice to those involved in the 
management and delivery of e-assessment within qualifications. It covers two key 
aspects of e-assessment: the management and delivery of e-testing, and the use of 
e-portfolios for assessment. The guide sets out the roles and responsibilities for 
those involved in e-assessment and presents a number of United Kingdom case 
studies. The toolkit that builds on the guide was designed to provide practical 
guidance on e-assessment and its role and impact on e-assessment practitioners. 

According to this guide, the critical success factors for e-assessment include 
processes that have demonstrable consistency and reliability. Staff must have the 
appropriate skills to manage and deliver these processes. In particular, staff involved 
in e-assessment must have a general understanding of the principles of fair 
assessment; an understanding of the importance of security in the conduct of 
assessment; a general familiarity with the e-assessment systems and delivery 
platform in use; and an awareness of legislation and codes of practice relevant to 
their institution, the general regulations of relevant awarding bodies, and regulatory 
authority guidelines and codes of practice. Finally, institutions must be able to 
demonstrate their accountability for the quality of these processes to external 
agencies. However, there is a degree of flexibility so that these processes and skills 
are able to evolve locally in response to technological improvements. 
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Methodology 

Sample 

In total, 48 individuals assisted in the study through interviews, teleconferences and 
small group discussions. The list of participants is provided in Appendix A. 
Participants were expert practitioners, expert auditors and others who had in-depth 
experiences with e-learning and audit in the VET sector. Participants were located 
through network sampling, and so through the recommendations of other 
participants. In addition, others who were interviewed were identified through the 
assistance of the Framework. Also assisting the project was the advice of a project 
steering committee whose members were Reg Allen (Tasmanian Qualifications 
Authority) and Vivienne Blanksby (Australian Flexible Learning Framework). 

The purpose of this sampling strategy was to locate individuals who were perceived 
to have considerable experience and expertise in the VET sector as either 
practitioners or auditors. The final sample included practitioners from all states and 
territories. Most practitioners had operated over time in multiple roles in the VET 
sector. These roles included as educational designers, teachers, trainers, assessors, 
assessment writers, curriculum designers, e-learning assessment objects 
coordinators and developers, online facilitators, and as internal auditors. 

The auditors were currently full-time professional auditors working either for 
government departments or as independent consultants. The vast majority worked 
across state and territories as external independent auditors, and had experience 
auditing and consulting in Australia and some overseas. As consultants most were 
contracted to complete audit work for multiple bodies across the VET and other 
sectors, and included audit projects for state government departments, RTOs, NARA, 
CRICOS and audits for the secondary school sector. 

Interviews 

Interview questions are provided in Appendix B. These semi-structured interviews on 
average were 60 minutes in length. All participants who completed the interviews 
were asked this same list of questions. However, the semi-structured format allowed 
for probing and additional questions. 
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Findings: Part A - key issues 

Formative and summative assessment 

Practitioners and auditors alike believed that e-assessment was finding its place in 
the VET sector as a form of diagnostic and formative assessment. They reported on 
examples of its use for diagnostic assessment of learner knowledge and skills at the 
outset of a qualification. The major use was for formative assessment that provides 
developmental feedback to a learner on his or her current understanding and skills, 
and helps change their performance through this feedback. Formative e-assessment 
as 'assessment for learning' was mostly focused upon giving feedback to help 
learners to continue to develop a unit of competence. 

The primary use of e-assessment for summative assessment was rare. Significantly, 
there was little support from the practitioners or auditors interviewed for the use of 
e-assessment as the primary form of evidence to support the signing-off of a formal 
qualification or certification of a skill. As one practitioner explained, the current 'state 
of play' is that practitioners and auditors are confident in the use of the online 
environment for formative pieces of assessment. On the other hand, both groups are 
very reluctant to sign-off on summative assessment that predominantly uses online 
tools. Many reasons were given, but most centred on the need for a variety of forms 
of assessment to gather multiple forms of evidence in performing a skill. 

Significantly, there has been considerable investment in Toolboxes to assist 
practitioners in exploring opportunities around e-learning. Our interviews revealed 
widespread knowledge of Toolboxes among both practitioners and auditors. Many of 
the expert practitioners interviewed had extensive knowledge in the use of Toolboxes 
and learning objects, such as the scenario based learning and assessment tasks. A 
few practitioners had actually prepared or were currently preparing Toolboxes with 
resource development teams for specific qualifications. 

Toolboxes were attractive as they were known to be heavily quality assured 
products. In addition, they had proved to be a very cost effective set of learning 
resources that were frequently purchased and adapted for local use. However, while 
it was accepted that they were not designed for assessment tasks, they were being 
used for formative assessment. Moreover practitioners were slowly becoming more 
aware of the potential for Toolboxes around e-assessment, especially as Toolboxes 
do involve projects that can be completed online. But this potential needed to be 
more fully promoted to VET practitioners. 



Applying quality standards 

From the perspective of auditors, the majority of audits confirmed that quality 
assessment practices were occurring. They cited many examples of institutions 
where the standards were applied as rigorously or even more rigorously to e-learning 
and e-assessment, as traditional training and assessment. For example, the delivery 
and assessment strategy was prepared and saved on Institute quality drives, with 
version controlled resources, and signed off. E-assessment tools were saved on the 
quality drive and once uploaded onto the website the last version was archived. 
There were also significant uses of hyperlinks to different Toolboxes, learning objects 
and activities that we could be used by teachers. On the other hand, some auditors 
have found that the quality document process is less likely to be followed in some 
institutions for online delivery than for more traditional forms of assessment. 
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According to our expert practitioners, at the core of some of the problems around the 
quality e-assessment is that some teachers assume that everything to do with 
e-assessments is located online. As a result, teachers do not prepare a study guide, 
a unit assessment plan or the assessment marking criteria. A number of practitioners 
and auditors reported a 'set and forget' attitude around some forms of e-assessment. 

Several practitioners reported that teachers at their institutions were required in 
audits to print off lengthy hard copies of online assessment materials. In their view, 
these auditors were not knowledgeable about e-learning, learning management 
systems or recent developments in assessment and e-assessment. In response, 
auditors explained that on occasions they had requested hard copies of online 
assessments for further review. This was not because they were 'out of touch' or 'not 
IT savvy', but as they had doubts about the maintenance of quality checks and 
procedures. They had serious concerns about the quality of the training and 
assessment materials, and found evidence that online students were being given 
access to second rate training and assessment. 

Many auditors talked about the warning signals sent by evidence of higher dropout 
rates in the online mode. High drop-our rates sent strong messages about training 
quality and assessment standards. In audits they found that some institutions failed 
to investigate and analyse drop out results to determine the primary causes. When 
such examinations were completed, students did not complete for multiple reasons. 
Factors included poor e-assessment practices, a lack of online support, and students 
not having the underpinning skills and support around computer literacy to engage 
fully in the online learning material. 

Many institutes are successfully managing such factors around learner knowledge 
and motivation with e-learning and e-assessment. At Brisbane North Institute of 
TAFE, the Certificate IV in Veterinary Nursing (see the case studies attached to this 
report) has evolved from a totally face to face delivery with traditional assessment, to 
the greater inclusion of more e-learning and blended delivery with the increased use 
of online forms of assessment. These students were described as 'digital natives'. 
Many did not have ready access to good quality computers, and some students had 
low levels of computer literacy. In response to such issues, special attention was 
given to improving the computer literacy of students and in resolving problems in 
accessing computers. The teaching team still accept hand written work and they 
have explored the use of web conferencing and online chats to better support these 
learners. 

In another example, the Gold Coast Institute of TAFE has set up a separate student 
site that enrols every student. The site has information about how to get online and 
the required technology to access to the library, literacy and numeracy programs, 
and online orientation programs. 

There are also operational and resourcing issues around computers. Few institutions 
have sufficient computers in a large enough space to have a majority of students 
undertake an online examination at the same time. The University of Tasmania (see 
case study reports) allows students to bring their own, since the majority have 
laptops registered on the UTAS wireless network. A related issue is equity, as some 
computers are more modern and have greater capacity. To manage this problem, the 
teaching team provides learners with the same operating system and application 
programs. To prevent students colluding with each other, the networking drivers are 
stripped out of the operating system provided. 

Maintaining confidence 

Respondents were positive about the operation of AQTF 2007. As also shown in the 
Formative Evaluation of the AQTF 2007 report completed by KPMG, there is strong 
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support for the intent and outcomes-focused principles of the AQTF 2007 21 . The 
outcomes-focused approach being adopted in the AQTF was well understood and 
appreciated by the practitioners and auditors we interviewed. Practitioners believed 
that the principles of continuous improvement were being applied by internal and 
external auditors who had audited their institute's programs. 

It was believed that the Essential Standards for Registration was well understood by 
those working in the VET sector. In particular, it was well known that assessment 
should meet the requirements of the relevant training package or accredited course, 
while the principles of assessment and the rules of evidence must apply. 

However, there was general agreement that practitioners and auditors were still 
coming to grips with the newer forms of assessment. Irrespective of industry, 
organisation, qualifications or the types of learners, there is a larger range of 
assessment strategies, processes and instruments to be managed, and more 
complexity in the evidence captured by these new forms of assessment to prepare 
for a successful audit. 

Very skilled practitioners were having their professional judgements around 
e-assessment challenged in audits. Auditors interviewed were critical of the capability 
of some practitioners to make quality assessment judgements. Overall, VET 
practitioners needed to be more skilled at meeting AQTF requirements with greater 
attention to the continuous improvement requirements of the AQTF. Interestingly, 
similar views were presented by practitioners about the knowledge and skills of 
auditors around assessment and e-assessment. 

At present, tensions do exist between practitioners and auditors around various 
forms of e-assessment. While valid, reliable, fair and flexible assessment is the 
cornerstone of the Australian VET system, the increased focus on the use of new 
technologies around delivery and assessment has resulted in some tensions. These 
tensions are on the one hand around maintaining the reliability and validity of 
assessment, and on achieving flexibility and cost efficiencies, on the other hand. As 
one respondent explained: 

"Technologies will support assessment in VET, and the system needs to support you so 
you are willing to give it a try, despite the occasional blow to ones confidence. In my 
first year of teaching, educational technologies meant classroom radios tuned to the 
ABC School of the Air, the rare Super 8 movie, and strictly rationed spirit masters, 
churned on a drum through mentholated spirits to make multiple handouts. My 
students' learning highs came mostly from sniffing the paper. Of course, if you made a 
mistake on the master, you'd start again or it would be there for all to see. Times have 
not changed that much really. We still need to be willing to learn by doing, to make 
errors with these new technologies, but to build your confidence by sharing that 
learning around new and improved assessment practices as you move on. " 

If there is a central factor at work according to those interviewed, it is all about 
maintaining confidence in any assessment system. Due to negative experiences in 
audits, some practitioners and their institutions are confused around the expectations 
and decisions of auditors around e-assessment. A few institutions with training 
programs that cross jurisdictional borders have received inconsistent rulings by 
different auditors in different locations. 

The current lack of confidence among some practitioners was seen in behaviours 
such as over-assessment and underassessment. Auditors reported examples of 
practitioners struggling with the concept of collecting sufficient evidence that is not 
'wheelbarrows full', as one auditor described it. There is a lack of knowledge, skills 
and in turn confidence among many practitioners around appreciating the time, effort 



21 KPMG Risk Advisory Services (2008) Formative evaluation of the implementation of 
Australian Quality Training Framework 2007. Completed for DEEWR Canberra. 
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and resources needed to develop e-assessment tools that capture the required 
evidence. In summary, an auditor explained: 

"There is a lack of confidence in anything to do with assessment whether traditional or 
e-assessment. This lack of confidence is caused by many factors. But they include the 
large turnover in VET of staff, more casual and part time staff, poor practices learned 
from others, a lack of good TAA training around assessment, and a lack of experts in 
the field of assessment. Often professional development workshops are conducted by 
people who don't have their own best practice guidelines in place. We need good 
worked examples of e-assessment representative of its various forms that help to close 
this gap that is emerging between practitioners and auditors around what are valid, 
reliable and authentic forms of e-assessment. " 

In short, auditors believed that they were very consistent in the messages they gave 
in their meetings with practitioners. Their key messages were that the: 

• same rigour applies to audits of all forms of assessment 

• onus is on the VET teacher to provide objective evidence that competency 
judgement is sound 

• system's integrity and user-friendliness needs to be promoted, and the 
practitioner must be prepared to demonstrate how the system ensures high- 
quality, industry-relevant training outcomes. 

Auditors emphasised that the principles of quality assessment apply equally to 
e-assessment. These principles are the same for the e-learning environment, as they 
are in traditional forms of delivery and assessment. These principles for the VET 
sector, set out in the AQTF 2007, include the need for assessment to be valid, 
reliable, fair and flexible. In addition, not only are methods of assessment flexible, but 
flexibility can also apply to the process. However, flexibility cannot be allowed to 
impact negatively upon quality. Too much adjustment beyond that which can be 
considered reasonable adjustment ultimately impacts upon the validity of some 
assessment processes and outcomes. 

Also possibly at the core of these tensions is a paradigm clash. The new paradigm 
that is being promoted around the nature of training is not supported by an old 
paradigm around the nature of assessment. The current system has attempted to fit 
traditional methods of assessment to a new training agenda that demands more on 
the job and therefore, more flexible delivery and assessment. Greater emphasis on 
building high level assessment knowledge and skills amongst practitioners and 
auditors alike would go some way to solving this issue. 

For example, this parading clash shows up in the growth in e-portfolios. A new 
training agenda that requires more RPL (recognition of prior learning) and more on 
the job assessment is finding that new approaches like e-portfolios provide more 
effective and flexible ways to gather evidence of competencies than more traditional 
forms of assessment. However, as noted later, there are also tensions here around 
the need to provide practitioners and auditors with better guidelines and examples 
that will build skills and confidence around the use of e-portfolios. 

Creating more acceptability 

Neither practitioners nor auditors believed that e-assessments would ever be the 
assessment of first choice for all VET qualifications. E-assessment will only continue 
to be the mode of choice when the training cannot be offered at all or only in a very 
limited way due to costs, learner location, learner type or industry training 
requirements. For instance, it is expected that e-assessment will be most popular, as 
it is at present, for training and assessment solutions that involves large numbers of 
learners who can sit e-examinations. 
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They envisaged considerable growth around the incorporation of various forms of 
e-examinations, short e-quizzes during training, and multiple choice questionnaires to 
test knowledge. In contrast, interviewees did not expect to see any considerable 
growth around the application of e-assessment around complex high stakes skills. 
The barriers to this uptake were technical, administrative and financial. 

Many interviewees emphasised that any reduction in the quality of assessment will 
impact on the longer term acceptability of e-assessment in the training sector. One 
practitioner with a long term interest in teaching and learning issues summarised 
their concerns: 

"To win the hearts and minds of teachers, industry and learners, we must achieve 
pedagogic validity. By this I mean we must use e-assessment to actually drive 
improvements in our assessment standards rather than just expect e-assessment to toe 
the current line. It must become part of our training system where learning and 
assessment are aligned. A big part of this will be getting the auditors more on side so 
that we can make continued progress to removing poor assessment practices generally 
by some practitioners, and more agreement about what is acceptable e-testing 
methods that are judged to be valid, reliable, appropriate and fair." 

Both practitioners and auditors noted problems in converting traditional assessments 
into electronic forms, where problems around validity or reliability are transferred or 
even exacerbated in the move from one medium to another. Even when the 
traditional forms of assessment are valid, converting them into e-assessments can 
invalidate them. As summarised by one practitioner: 

"Today, educational technologies range from Web 2 worlds, high -investment, 
sophisticated, integrated, organisational-wide learning content management and 
learning management systems and computer networks, to more personal and portable 
devices and applications like, "smart" pens, mobile phones, MP3 players, laptops, 
PDA's and notebooks. Mistaken assessment approaches though are still there for many 
more to see and still mean starting over. Technologies can enhance and extend 
possibilities, but they don't transform the nature of assessment itself. Just moving poor 
assessment design online, for example, will never make it better. " 

It was felt that such issues require more discussion. Improvements in the quality and 
use of technologies will allow practitioners to move to more authentic forms of 
performance assessment that is currently difficult and too costly with older 
technologies. For example, assessment that combines text, images, sound and video 
will make e-assessments ranging from online tests to simulations more realistic and 
interactive. 

This higher level of integration between teaching, learning and assessment through 
the use of technology, as many have argued in our interviews, will see the final stage 
of the transition in the VET practitioner role. Technology will assist the move from a 
still predominantly teacher-centric position, to VET teachers being facilitators of 
student learning. Assessment will continue a transition from being given, done alone, 
book or manual based, done in class, and teacher assessed; to being negotiated, 
done collaboratively, text/audio/video based, done anywhere, and teacher, self- and 
peer-assessed. 

Auditor experiences with e-assessment 

Auditors reported that the forms of e-assessment currently being audited were not 
large. The dominant e-assessment methods being audited included on line tests to 
assess knowledge, and the use of blogs and wikis to promote reflection and self- 
assessment. The use of e-portfolios was on the increase, but evidence from 
e-portfolios was not a frequent part of an audit. Virtual world scenarios were rare. 
Most practitioners known in their institutions to be the 'e-champions' were found by 
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auditors to be using a mix of face-to-face assessment strategies, combined with 
online assessment. In most cases, this online assessment was the multi-choice test. 

Auditors reported that their position in any audit is not to hold any preconceptions 
about the form that evidence may take. Auditors emphasised that the majority of 
them are not ICT experts. Despite this, some feel that practitioners in audits have 
tried to baffle them with the new technologies or by 'technological speak' to get 
around questions that challenged aspects of the evidence. However, regardless of 
the evidence gathering methods used, auditors reported that they want to see 
evidence that the training and assessment is meeting clients' needs, the principles of 
assessment and rules of evidence, and the assessment is aligned to AQTF level. 
They want to be able to confirm dimensions of competency, employability skills and 
transferability. This message was a very consistent one in the auditor interviews. 

There was considerable agreement among auditors about the common risks, as 
some described them, around the current use of e-assessment. These risks were 
summarised in this auditor's comments: 

"/ have a number of concerns that need to be addressed. How are they managing the 
security of electronic data? I want confirmation around the authenticity of students' 
work, and the access that students have to ICT hardware/software so that they are able 
to complete their training and any online assessment. In particular, as with any form of 
assessment, I will be asking about how they are managing the concerns I typically raise 
around how a sufficient range of evidence is being gathered, issues around 
employability skills and transferability, and overall, how the progress of students is 
being monitored. " 

Some auditors highlighted how the demands upon their time possibly encouraged 
their decisions around the validity of assessment and e-assessment to fall on the 
side of caution. Some auditors believed that they did not have sufficient allocated 
paid time to examine the e-assessment methods as thoroughly as they would like. 
Expected to audit four or five qualifications in a day, as well as the supporting 
management systems, meant that at best they only gained a snap shot of the 
e-assessment being used. 

Also some identified other personal factors at work in making them wary of 
e-assessments. By their nature, as a number of auditors explained, they are 'natural 
sceptics'. Many auditors believed that they were 'naturally suspicious people', and 
what they had seen around the poor use of e-assessment around authenticity 
supported their inclination to be cynical and suspicious. Others noted that despite 
considerable opportunities for professional development, many auditors were 'still 
wedded to tradition and paper evidence more than we should be'. Current auditor 
training did not provide wide exposure to e-learning or e-assessment systems. 

Issues around validity 

Validity was a major concern for auditors. While they accepted the benefits of using 
new mediums to give an extra dimension to VET assessment, they were seeing poor 
assessment practices at work. E-assessments were not validly assessing the skills 
being tested and the assessment did not always address the intended learning 
outcomes. 

Some e-assessment had been introduced without establishing learner needs and 
completing training needs analysis. In occupational health and safety (OH&S) and 
blue card training using online tests, auditors reported learners being 'trained to 
complete the assessment' through poorly designed tests and poor test procedures. 
Other online assessment tools were found not to be current and had not kept pace 
with changes in legislation. 
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To keep pace with new legislation, the Sunshine Institute of TAFE has a 
development site that is accessed by key people before it goes live. Their critiques 
assure that the traditional or e-assessment is meeting the respective AQTF level. 

Asked what might be considered to be reasonable efforts to validate assessment, 
auditors cited the need for greater use of multiple assessments, independent 
validation from subject matter experts, and the option of a mandatory interview at the 
need of the learning and prior to judgements about competency. The Department of 
Defence is seen as exemplary here. In what is seen by auditors as an industry that 
engages in high stakes learning, a nationwide group of subject matter experts are 
used to complete mandatory interviews before signing off on qualifications. These 
qualifications are gained through a mix of e-assessment (e.g. online tests, scenarios, 
virtual reality, simulations) and more traditional assessment. Defence also stipulate 
that a person who trains an individual is never allowed to complete their summative 
assessment. 

Auditors wanted to see better practices around evidence gathering through multiple 
sources such as observations, testimonials and work-based assignments to support 
the validity of the online assessments. Some RTOs were completing random 
sampling of learners to validate evidence from both traditional and e-assessments. In 
addition, they mentioned the value of improved validation through improved 
evaluation and piloting during the design and implementation of e-learning and 
e-assessment tools. A number of auditors were critical of the lack of attention by 
practitioners to piloting online assessment materials, and in their view they often 
found too much material placed on the learning management system without too 
much thought or review by practitioners. 

Issues around authentication 

Audits also revealed issues around delivery that included poor security, unreliable 
infrastructure that impacted upon the user in completing online assessment, and 
issues of accessibility and ease of use. At its worst, auditors found some 
e-assessment driven by cost cutting measures by the institution rather than by any 
benefits to the learner. Also some training providers were not keeping on top of some 
users who were skilled at cheating by finding shortcomings with the online 
technologies being used for assessment. 

Practitioners reported upon various technological solutions to deal with challenges to 
the authenticity of evidence gained through e-assessments. These strategies 
included the use of security passwords; online authenticity statements completed 
prior to being able to submit or upload assessments; electronic signatures linked to 
statutory declarations; the use of timed quizzes; and the greater incorporation of 
adaptive testing where questions are determined by prior responses. Other 
technological solutions lock out the critical functionality during an on line test to stop 
collaboration between students. In addition, new technologies can encrypt data files 
so the program being used in the assessment can only run on computers with a 
pre-registered network card. 

It was widely accepted by those interviewed that the right technology can enhance 
the security of e-assessments. Indeed, there are an increasing number of computer- 
mediated products and tools on the market that can restrict the examination 
environment to being a closed one (e.g. Pearson VUE; Articulate Quizmaker 2.0, 
MaxExam, BrainsBuilder, Secureaxm). The growth and success of these products is 
seen as promoting the use of e-assessment examinations for the so-called higher 
stakes written examinations. 

A leader in the use of technology to craft well-designed and executed e-examinations 
is the University of Tasmania (see case study). Its lecturers are using an open source 
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operating system on a live CD to provide a holistic examination environment. 
Learners are shown how to use the e-examination system during a practice tutorial to 
ensure they can operate the software. The system allows learners to practise 
personal skills at home without copyright infringement. It also permits e-examinations 
to be supervised by invigilators without specialist information technology skills 
through displaying a unique desktop background image for security. E-examinations 
are constructed for a range of software environments, with candidates working on 
isolated workstations or 'open book' contexts with full internet connectivity. 
Candidates can use their personal laptop without gaining advantage because local 
disk access is blocked. 

Practitioners were also aware of developments in other countries with trials around 
finger print technology, and the use of technology that matches photographs on file 
with the person doing the online assessment. Some online systems, for example, are 
authenticating the individual undertaking the assessment through photographs and 
video checks at random intervals. Many of these features can be set up now, or in 
the near future, in the current functionality of the learning management systems. Also 
as another check, students can be supervised in the online testing environment, as 
might be done in more traditional classroom assessment. 

Auditors were in favour of technological solutions to managing the authenticity of 
e-assessment to make sure that the assessment was completed by the actual 
learner. However, their position, shared by many practitioners, was that those 
students who want to cheat will always be one step ahead. What is required are 
increased efforts by teachers to build and maintain effective, high quality and 
supportive relationships with their online learners, as well as keeping regular contact 
with employers. Such relationships assist teachers in knowing that the e-assessment 
was completed by these learners. 

Another exemplar in the use of e-assessment practices that combines technology 
with relationship management is Blue Dog Training. It has developed innovative 
flexible e-learning resources and state of the art e-assessment that uses online bite- 
sized 'chunks of learning'. At the 2009 AustralianTraining Awards, Blue Dog won the 
Small Training Provider of the Year and the Innovative Business Award. Blue Dog 
has custom-designed interactive, animation-rich learning resources and uses online 
quizzes for learning and assessment. An integral part of the assessment process is 
an electronic skills profile, like a form of e-portfolio, that documents photographic and 
journal evidence of learners' practical competencies. Assessment methodologies 
consist of an online series of complex exercises which test the underpinning 
knowledge of all competencies. 

Blue Dog trainers work closely with on the job supervisors in gathering evidence on 
all practical tasks. Trainers verify the onsite evaluations by questioning not only the 
apprentice but also the supervisor. Assessment is further authenticated by recording 
the frequency of all tasks, as well as photographic evidence, where applicable. Blue 
Dog report that apprentices are not deemed competent until all aspects have been 
thoroughly understood and implemented. Besides the recent successes at various 
training awards, Blue Dog Training recently completed a successful audit that will 
allow it to deliver Certificates I, II and III in the area of engineering based on the 
same successful delivery and assessment model used for carpentry. 

In summary, any form of assessment raises concerns over malpractice. However, 
Blue Dog Training demonstrates that building and managing effective working 
relationships with learners and their employers is as important as applying the latest 
in technological solutions. Closing on this point, a practitioner shared the following 
reflection: 

"We had the same issues raised when we introduced distance learning. Is this the 
students' work? In fact, the same concerns exist around traditional assessment. How 
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do I know that written report is the student's work? I know because it is one piece of 
several pieces of assessment. I know in particular because I have developed a 
relationship with that student. I talk to them by mail and phone, and also I talk to their 
employer when I can. Any good teacher seeks such validation. " 

The online quiz 

The standout example of e-assessment is the online quiz. Practitioners and auditors 
reported that online tests are commonplace in the assessment practices of many 
VET organisations. Such tests are largely used for formative assessment rather than 
as major contributors to any summative assessment. In many instances, the online 
quiz is conducted using an institutional learning management system such as 
BlackBoard, WebCT or an in house product. At its best, the online quiz involves 
video in questions, particularly for scenarios in authentic assessment; adaptive 
testing where the next question to be posed is determined by a prior response; and 
large and varied question banks with randomisation of questions and response 
orders to reduce cheating. 

Quality assessment enhances the learning process, as well as providing final 
confirmation of competency outcomes. Practitioner users of quizzes cited several 
advantages for the learner and for the teacher. Advantages cited included ease of 
use around preparation and marking; ability to assess learner knowledge quickly; and 
to provide regular and often instant feedback. However, disadvantages included 
poorly conceptualised and constructed on line tests that had limited validity and 
reliability because they were not assessing the knowledge or skills required for the 
competency; and little history of revision, updating or even basic psychometric 
analysis of the nature and pattern of responses to ascertain levels of validity or 
reliability. 

According to most auditors and many of the expert practitioners, some teachers have 
little or no understanding of how to design valid and fair online quizzes. Teachers 
underestimated the skills and effort needed. These skills include being able to 
construct feedback that is front-loaded; starting with easier questions and making 
later questions more difficult; checking assessments with subject matter experts and 
high performers; identifying weak questions and improving or eliminating them; 
reviewing the content of a question to ensure the identified knowledge is covered; 
running basic psychometrics to examine if the question items differentiate between 
better and poorer students; and managing security issues by writing multiple banks of 
items with different orders and choices depending on the answers given. 

As a very experienced e-learning practitioner noted: 

"There are plenty of bad examples of poor use of technology for e-assessment. One 
that comes to mind is a system used by a national body. All a user has to do to 
complete the course is put a Vend' in the address bar and the course is complete. The 
course is not too bad and the assessments are okay but these types of shortcomings 
give the whole online training industry a bad name. " 

Unfortunately, auditors cited numerous examples of such poor practices. At worst 
online tests and short quizzes were described, by a number of auditors and 
practitioners, as 'cheap options'; and 'ill-thought out efforts by the provider to make 
money quickly or to save money'. As explained by one auditor: 

"As an auditor I see e-learning and e-assessments being used to support economies of 
scale rather than a methodology designed as a result of effective training needs 
analysis that would ensure targeted support systems for appropriate learners." 

Indeed, some auditors believed that the practices of some institutions almost 
demonstrated a belief in two types of learners - those in class who got face-to-face 
contact with many of the best teachers who provided complex and multiple forms of 
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assessment; and those online learners who got access to downloaded notes, PDF 
files under the guise of e-learning, and poorly designed and repetitive online quizzes. 

Some very experienced and highly successful e-learning practitioners interviewed 
believed that many learners have never really experienced well designed e-learning 
and e-assessment: 

"Good assessments do not come cheaply, and many institutions are not prepared to put 
the time and money into creating these assessments. They just want to get content and 
assessments up online as quickly and as cheaply as possible. A well constructed online 
activity including content and assessment is time consuming. It is expensive and 
requires a great degree of skill from content experts, multimedia persons and 
assessment writers who know and understand the tools at their disposal and how to 
use them effectively. Unfortunately, many students and their employers have yet to be 
really exposed to top class e-learning and e-assessment. " 

Assessment support materials 

The Quality of Assessment Practices Stage 2 report completed by the NQC identified 
important gaps in assessment support materials that need to be addressed to build 
assessor capability 22 . The report noted a very limited range of up to date assessment 
materials that address many emerging issues that are impacting on quality 
assessment. Relevant to this examination of the AQTF and e-assessment, the NQC 
report found a lack of materials that provide practitioners with high quality advice on 
peer, self and negotiated assessment, on formative assessment and in the use of 
simulation. 

In another stock-take of assessment resources, it was found that there were few 
professional development resources aimed at improving VET assessor skills in 
formative assessment 23 . On the other hand, the use of more formative assessment is 
predicted to expand in the future. In addition, the same stock-take revealed that the 
vast majority of resources are web-based, but VET assessors were least aware of 
these web-based resources. 

Possibly due to the rate of growth in the uptake of e-learning, our respondents 
recognised gaps in assessor support materials around competency-based 
assessment in these new online learning environments. A major concern raised 
across interviews is the plethora of poor quality e-assessments tools being used 
online, on intranets and in blended learning. Both practitioners and auditors noted 
that either due to inexperience or a lack of knowledge, some teachers fail to identify 
such poor quality tools as they select their online assessment. In turn, auditors see 
the consequences of these poor choices in their audits. 

The value of e-portf olios 

There were very positive views about the current and future roles for e-portfolios. 
Practitioners and auditors reported slow but persistent growth in the use of 
e-portfolios in training organisations. An e-portfolio is defined as a learner-driven 
collection of digital artefacts articulating experiences, achievements and evidence of 
learning 24 . As those who use e-portfolios point out, while there are many different 



22 National Quality Council (2009) Quality of assessment practices. Stage 2 report. Canberra: 
DEEWR: 

http://www.nqc.tvetaustralia.com.au/ data/assets/pdf file/0009/50976/Quality of Assessme 
nt Practices - Stage 2 Report.pdf 

23 Mitchell, J., & Ward, J. (2009). Resources for building VET practitioner skills in assessment. 
Brisbane: Department of Education and Training. 

24 JISC (2008) Effective practice with e-portfolios Supporting 21st century learning: 
http://www.iiscinfonet.ac.uk/e-portolios 
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definitions of what an e-portfolio is or could be used for, in part this flexibility of 
definition has supported the considerable growth in their use. 

Practitioners interviewed noted that e-portfolios can streamline evidence and 
validation, while the authenticity of evidence can be verified by appropriate learning 
management systems/strategies. E-portfolios are proving to be a valuable support in 
enabling learners to document and reflect upon their learning and skills development. 
Among the uses that are most relevant to e-assessment is their utility in recording 
evidence of employability skills and in the verification of competencies and 
qualifications. E-portfolios were seen by the practitioners interviewed to be providing 
learners with new ways to present evidence from formal and informal learning 
environments. For some learners, the ability to electronically verify evidence is 
streamlining applications for employment, course admissions and RPL. 

Some auditors, however, had concerns about e-portfolios as audits had raised 
serious questions around the authenticity of evidence gathered by learners and the 
validity of the assessment tasks given for units of competency. Practitioners too 
reported being reluctant to embrace e-portfolios, particularly due to the lack of 
guidelines and good examples around structuring suitable pieces of assessment. 
They also had negative experiences where learners have assembled overly large 
collections of evidence which were difficult to assess. 
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Findings: Part B - next steps 

Improving auditor consistency and training 

Practitioners appreciated the cross-jurisdictional experience that some auditors now 
brought to the audit, as many auditors have experience undertaking audits in one or 
more jurisdictions, including those associated with NARA Such experiences were 
useful in helping both practitioners and auditors become more familiar and confident 
in recognising good practice around assessment and e-assessment. 

Practitioners argued that auditors should have similar approaches and reach 
consistent decisions. But this was not always the case. Some training providers had 
experiences in designing and delivering training across multiple jurisdictions that 
involved the use of the same forms of e-assessment. In a few cases, institutions 
found that forms of traditional and e-assessment signed off by auditors in one state 
or territory were problematic to auditors in other states or territories. There were also 
experiences of different audit outcomes for NARA and other audits within the sector. 

Many examples were given from auditors of frequent dialogue between jurisdictions. 
However, a number of auditors put forward the view that the ways in which auditor 
engagement/training occurs is as variable as the number of jurisdictions. There are 
several mechanisms used by jurisdictions to provide professional development of 
auditors. Notable examples included frequent teleconferences on specific matters 
and annual conferences (e.g. NQC) that hold discussion of key and emerging issues. 
The 'trickle down effect' as it was described was the main way in which dissemination 
of good practice occurs among auditors. There are also occasional auditor 
exchanges or audit teams with auditors from several jurisdictions occasionally 
auditing together. 

While additional standards might be issued around e-assessment, most auditors 
believed the current AQTF standards were sufficient to support quality e-assessment. 
As many auditors remarked, if we are to provide a set of standards around 
e-assessment, why not also provide standards around RPL, another area of concern 
that many auditors had regarding RTO performance. However, they were much more 
supportive of guidelines and examples of best practice around e-assessment. 

A number of ideas were put forward by our expert auditors about how to promote 
better levels of consistency among auditors. One strategy is to allocate the task of 
training auditors to a third party with no direct interest in the outcome. In practice this 
would not only cover e-assessment, but everything to do with being an auditor under 
the AQTF. The Standard advises on the steps you would take to achieve auditor 
consistency through this third party approach. It was proposed that this approach 
would mean that there are no hidden agendas or unwritten expectations. Key issues 
of concern would be agreed to by representatives of industry, regulators and other 
stakeholders. In Australia there are currently two bodies with this third party 
capability: RABQSA 25 and International Register of Certified Auditors (IRCA) 26 . 

Generally practitioners believed that minor inconsistencies and issues in audits were 
quickly resolved with the auditor, quality consultant and the RTO. To improve audits 
around new issues like e-assessment, there was some discussion about the need by 
auditors or auditing organisations to engage RTOs at an earlier stage than at 
present. Many practitioners interviewed called for developing greater consistency in 



25 See http://www.rabqsa.com 

26 See http://www.irca.org/auditortrain/auditortrain.html 
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auditor judgements through regular auditor validation and moderation sessions and 
workshops. Such activities would offer the opportunity to work through key issues 
which have caused confusion among auditors and practitioners. Practitioners also 
wanted to learn more about how auditors were keeping track of recent developments 
in areas such as e-learning and e-assessment. Auditors also wished to learn more 
about the professional development activities around e-learning for practitioners. 

Our auditors were very up front about the need to continue to broaden their 
understanding of e-learning and e-assessment. As said a number of times by 
auditors in the interviews, many of the current auditors started in this role when there 
was little use of technology. Most auditors are now over 45 years of age. The length 
of their experience has allowed them to bring considerable knowledge and expertise 
to the audit process. In the vast majority of their audits, they are still dealing 
predominantly with assessment methods that they are familiar with from their own 
training experiences. However, many auditors are not very familiar or experienced 
with e-assessment in its various forms. As one auditor explained, his personal 
experience with e-assessment is limited to a single online self-assessment that he 
completes of his own performance each year. 

Auditors acknowledged that this new era of outcomes focused auditing and the 
greater use of technology in VET training and assessment made matters 'more 
complicated around what the standards are', as one auditor put it. Validity, reliability, 
authenticity and applicability were still the key concerns. However, many auditors 
admitted their thinking around assessment needs to be further challenged as it is still 
too linear rather than more holistic. 

More well trained and informed practitioners 

Many practitioners, and even more so the auditors we interviewed, proposed that the 
assessment knowledge and skills of VET practitioners must be enhanced. The 
current VET practitioner needs to be more able to: 

• possess the technical skills to devise or create valid, reliable and authentic 
assessment and e-assessment 

• demonstrate knowledge and understanding of assessment best practice 

• understand and address issues relating to the use of e-assessment 

• access e-tools or resources that are easy to use and cost effective 

• access high quality continuing professional development courses that allows 
them to become familiar with the principles and best practice of 
e-assessment. 

According to the expert practitioners and auditors we interviewed, a key message 
that needs to be better communicated is that the same educational rigor and 
standards must apply to all delivery and assessment methods. Just as demanded for 
traditional assessments, e-assessments must have learning and assessment 
strategies, unit assessment plans, marking criteria, model answers, and benchmarks 
against which performance is to be judged that meet the training package 
requirements. As many auditors have reported, however, this is not currently the 
attitude and practice in many institutions. 

An obvious starting point is the current Certificate IV in Training and Assessment 
(TAA40104), and improving knowledge and skills around unpacking training 
packages to design and develop more appropriate assessments for face-to-face and 
e-learning. The overwhelming majority of practitioners interviewed were highly critical 
of the lack of coverage to e-learning and e-assessment issues in the current 
Certificate IV. It is not until the Diploma that teachers and trainers complete units of 
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competency on online delivery. One practitioner described the situation around online 
delivery and assessment as 'an awkward point in history', although he was confident 
that the training for new VET teachers and trainers will catch up in time. 

Our expert practitioners interviewed were especially critical of the lack of validation 
and moderation processes and skills applied to both assessment and e-assessment. 
In response to such concerns, the Gold Coast Institute of TAFE is planning a site on 
its learning platform to support teachers in discussions and sharing of issues around 
'education integrity'. Teachers will validate and moderate on an open platform, so 
inviting the comments of other teachers. This process will supplement rather than 
replace existing moderation and validation processes. It is expected that this process 
will lead to more discussion on strategies for improving the benchmarks and the 
outcomes of assessment. 

Also the Great Barrier Reef Institute of TAFE have a planning calendar for the whole 
faculty for the year that has set months where teachers have to complete activities 
such as industry liaison, course review, assessment review, and updates to program 
information and brochures. In this set calendar, three months in the year are 
moderation months where teachers have to formerly get together and validate the 
assessment tools. 

Western Sydney Institute is developing the capability of teachers in using technology 
to support teaching, learning and assessment though its el_F program (see the case 
study reports provided as part of this research for more detail). The program is 
designed to allow teachers to experience a range of activities which include 
facilitation strategies and the use of emergent educational technologies. It is 
expected that better learner-teacher collaboration will build the platform for improved 
assessment practices and tools. The aim is to increase the professional skills and 
knowledge of staff in using new technologies. 

In particular, el_F assumes that the online environment requires practitioners and 
learners to establish stronger relationships with each other by using appropriate 
communication tools and skills. Tools used in the program include Wikispaces, 
Adobe Connect web conferencing, Moodle, blogs and a number of other web 2.0 
technologies. Also el_F graduates are encouraged to participate in e-learning 
activities away from the program. 

The need for worked examples of e-assessment 

Assessment strategies need to be selected to fit the learner's abilities and 
experience, and to match the outcomes that need to be achieved. However, auditors 
gave numerous examples of how traditional as well as online assessment tasks did 
not match the units of competency being measured. There were related concerns 
about the inability of practitioners to authenticate the learner's performance, to 
demonstrate learner involvement in the assessment process, and to explain how 
teacher-learner online communication was effectively managed. 

In response, practitioners want access to better examples that illustrate the use of 
quality e-assessment. They require examples from both practitioners and auditors 
around e-assessment that meets the requirements of the relevant training package 
or course, and how the e-assessment is conducted in accordance with the principles 
of assessment, the domains of competency and the rules of evidence. 

Communities of practice and teacher discussion forums potentially were seen to play 
a vital role in identifying and sharing good e-assessment practices. Indeed, the use of 
communities of practice is identified in past research as one of the sector's major 
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drivers for continued innovation 27 .The main purpose of these communities of practice 
would be to bring together internal and external auditors, practitioners, management, 
educational and instructional designers and others. In these forums, participants 
would assist each other in addressing current hot topics in e-assessment such as 
non-compliance, assessment validation and moderation and authentication. 

The need to formulate e-assessment guidelines 

As noted by numerous interviewees, the VET sector has not put enough thought into 
what the guidelines are for online delivery and assessment. Currently those working 
in the sector are confused. They are trying to make their processes appear to fit 
guidelines that are grounded in paper-based approaches to assessment. While many 
believed that both practitioners and auditors have come a long way in better 
articulating their thinking and processes around assessment, it is time to provide both 
groups with more assistance through e-assessment guidelines and good examples. 
As an auditor explained: 

"In the end we are coming in from the back end as auditors, establishing and 
demonstrating how the training and assessment are meeting the requirements of the 
training package, rules of evidence, and the units of competence. You can tell when a 
lot of thought has gone into the development of effective assessment systems. We see 
enough bad examples to warrant the value in capturing and sharing good worked 
examples that will help all of the parties involved". 

There was a strong support in our interviews for the compilation of good examples of 
e-assessment. Those interviewed were very positive about the current Users' Guide 
to the Essential Standards for Registration, and it was suggested that this document 
might be expanded with examples around traditional and e-assessment within its 
explanatory notes or in other sections. 

However, more respondents talked of the need for a separate handbook or resource 
that was a collection of examples that address all assessment, and includes 
examples in both traditional and e-assessment modes. There was also, but to a 
lesser extent, some support for the exposure of poor examples of e-assessment in 
the same handbook. More generally, there was widespread support for the 
establishment of guidelines around the wide range of activities where digital 
technologies are used in assessment, including the designing and delivery of 
assessments. 

Hand in hand with these guidelines and worked examples were quality documents 
and procedures to meet best practice to ensure that online delivery and assessments 
meet the AQTF standards. Examples of these documents might include project sign- 
off documents that provide AQTF assessment requirements; an online delivery and 
assessment matrix (i.e. a planning tool prior to developing any resources or 
assessments); and post delivery unit review and moderation documents (including 
online components). Such documents have been prepared by the NQC and in other 
countries as they move to preparing their own guidelines (see the NQC and United 
Kingdom's Ofqual websites (see page 10). 

Similar calls have been made in Australia for e-assessment guidelines and best 
practice examples for e-portfolios. The VET E-Portfolio Roadmapr' 8 was 



27 Callan, V.J. (2004) Building innovative VET organisations. Adelaide, NCVER: 
http://www.edu.au/research/proi/nr301 1 .pdf 

28 The VET E-portfolio Roadmap is a strategic roadmap for e-portfolios to support lifelong 
learning: http://www.flexiblelearning.net.au/files/VETePortfolioRoadmap web. pdf 
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commissioned by the Framework's E-portfolio 29 business activity to assist in the 
development of a national standards-based framework to support e-portfolio systems 
in the national training system. These national guidelines around e-portfolios will 
promote a more consistent understanding of the standards and provide evidence of 
good practices. 

Significantly, in the United Kingdom a set of national guidelines do exist for 
e-assessment. As mentioned earlier in this report, the E-assessment: Guide to 
effective practice was developed by their qualifications regulators to promote and 
support effective practice and quality improvement in the assessment of 
qualifications. The Guide also has links to an e-assessment toolkit. The United 
Kingdom guide offers a practical treatment of issues around the delivery of 
e-assessment through e-tests and e-portfolios. It describes the knowledge and skills 
needed by practitioners and others involved in the e-assessment process. 
Significantly, it is not a guide to the development or procurement of technology or to 
the technical design of the content of e-assessment. 

A criticism of this United Kingdom guide, however, is that it is highly focused upon 
e-testing, with some attention also to e-portfolios. There is little attention to the wider 
range of e-assessment practices and tools. Nevertheless, given the evidence in the 
current report around the poor design and use of online tests and quizzes, this focus 
is possibly not misplaced, and a good first step. 

A minority of practitioners called for a national body to coordinate activities to better 
support the building of teacher capabilities around e-assessment. This body for 
e-assessment would support VET practitioners, collecting best practice examples, 
guidelines and tools, and communicating the positive aspects of e-assessment. 

Next steps 

Within the VET sector, the objective is to develop and deliver valid, reliable and 
authentic assessment that demonstrates competence against industry standards. 
The AQTF 2007 sets out standards to ensure that assessment is conducted in 
accordance with well-established principles of assessment and the rules of evidence. 
However, we know from this report that even experienced teachers struggle to meet 
these standards in the traditional face-to-face modes of assessment, without the 
added complexities of achieving quality assessment in the new e-learning 
environment. 

In looking to the future, respondents were asked what would need to happen for 
e-assessment to become more mainstream. Given the nature of vocational 
knowledge and skills, few expect that e-assessment will be the assessment 
technique of choice for all qualifications at all levels. However, there will be a 
continued uptake of e-assessment techniques to the stage that some forms like 
online tests and quizzes will be a standard feature of VET assessment across all 
qualifications. The major drivers will be cost savings, ease of use and the potential 
for the provision of immediate feedback to students. 

However, as noted in the comments of practitioners and auditors, this continued 
evolution around what we understand in the VET sector as valid assessment will not 
be without its tensions. There will be understandable differences of opinion amongst 
those passionate about maintaining a quality VET system. Regulatory arrangements, 
standards and guidelines that are allowed to evolve over time will promote more 
debate around innovations than those that demand compliance. The ultimate goal is 



29 The Framework's E-portfolios business activity supports the development of national 
e-portfolio standards to improve the portability of learner-collected evidence of learning: 
http://flexiblelearning.net.au/e-portfolios 
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to build continued confidence around these new forms of assessment that will 
emerge as the sector and its industry clients embrace more flexible modes of 
delivery. 

Significantly, those interviewed agreed that the time is right to discuss what are 
appropriate guidelines and good practice examples around e-assessment. They 
agreed that no single group should lead this debate. Most practitioners and auditors 
accepted the value of pulling together a shared understanding of the key issues. 
Significantly, no person interviewed thought that the issues raised in this report will 
disappear. Rather they will increase as the online environment continues to be 
explored for ways of providing more flexible, customised and cost effective training 
solutions for all clients of the sector. In summarising the way forward, one practitioner 
suggested: 

"We cannot let the auditors dictate what we teach and how we assess online. We 
understand the pedagogy, and they don't. They must be more open to change. But 
practitioners have also been found out by audits that show that we do not fully 
appreciate the knowledge and skills required to design and to deliver effective online 
assessment. Practitioners need to better apply themselves to learning these new ways 
of assessing but they need the support of their organisations and more support around 
what is expected. " 

Interestingly, prior research into the assessment of more traditional forms of delivery 
and training in the VET sector has already identified numerous ways to improve the 
quality of VET assessment practices 30 . This earlier work can also guide the next 
steps around e-assessment. These developments around more traditional 
assessment include clearer and well-documented assessment policy within RTOs; 
improved evidence of how RTOs go about the assessment process; enhancements 
in validation and moderation processes in organisations to promote greater 
confidence in assessment outcomes; and improvements to the 'what' and 'how' of 
professional development for those involved in assessment in the VET sector. 



Concluding comments 



In conclusion, the current report builds upon this past work on assessment in the 
VET sector. Importantly, it also puts forward ideas that might promote a greater 
understanding by both auditors and practitioners of more specific issues in relation to 
e-assessment. 

Following on from the findings from our interviews, we propose further discussion of 
the following ideas: 

1 . Preparation of guidelines for e-assessment in the VET sector titled 
E-assessment: Guide to effective practice in the VET sector. This guide will 
also provide a definition of e-assessment for the Australian VET sector. The 
Guide is not intended to be mandatory, exhaustive or definitive but to be 
educative in nature. 

2. Collation of examples of good practice around the use of a wide variety of 
e-assessments. These examples will link back to the Guide document. 

3. Delivery of workshops that promote a broader discussion between 
practitioners, auditors and others of the key issues raised in this report. In 
addition, the workshops can assist in outlining the key aspects that need to be 
incorporated into the guidelines around e-assessment. 



30 Ibid. 
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4. Preparation of support materials around the design and delivery of online 
quizzes, including the measures for assessing the quality of multi-item 
question banks and for assessing, tracking, and reporting upon the quality of 
question items in item banks over time. 

5. The development of units of competency in e-assessment that become part of 
any planned revisions to the current TAA (training and assessment). 

6. An independent organisation for e-assessment that provides support for 
professionals in the field, collecting best practice guidelines and 
communicating the positive aspects of e-assessment. This body would also 
identify future areas of research around e-assessment, in line with similar 
bodies in the United Kingdom (e.g. JISC). 

7. Towards assisting in the preparation of the Guide and best practice 
examples, to complete a state-of-the-art review of techniques in 
e-assessment. This review will develop a picture of the breadth of advanced 
e-assessment techniques in use around the world, and undertake a technical 
and educational evaluation of a selection of the most promising opportunities; 
present short case studies of the most significant examples; and draw 
conclusions and raise the level of debate about how the most relevant of 
these could be applied to the VET sector. 
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Appendix B: Interview questions for expert 
practitioners and auditors 

E-assessment and the AQTF project: Victor Callan and 
Berwyn Clayton 

University of Queensland Professor Victor Callan, together with Berwyn Clayton from 
Victoria University, are conducting a national research project to examine what can 
be done to promote greater confidence in the validity of e-assessment in the 
Australian VET system. Funded and supported by the national training system's 
e-learning strategy, the Australian Flexible Learning Framework, this research will 
explore the views of expert practitioners, assessors and auditors on establishing 
practical recommendations for e-assessment processes that require auditing under 
the AQTF. 

The aims of the project in 2009 and into next year are: 

1 . Consultation with auditors and practitioners in 2009 (e.g. interviews with 
targeted and respected auditors, quality managers, practitioners) to build a 
picture of the key issues around how practitioners are using e-assessment 
and around the auditing of current e-learning assessment practices in public 
and private registered training organisations. The purpose of these 
consultations is to develop a common understanding of the key assessment 
and auditing issues, the challenges, and various strategies and approaches 
being employed 

2. Analysis of three case studies in 2009-2010 of e-learning assessment 
practices in the VET system (including cases that support the reliability and 
quality of the Flexible Learning Toolbox content) 

3. Consultation through workshops in 2010 with AQTF auditors to determine the 
requirements for draft national guidelines in relation to the auditing of 
e-learning assessment strategies. 

Questions: 

Part A. Your background and VET and other experiences 

Q1 . Can you please describe your current role as a practitioner, or auditor or other 
role and how it links to this issue of e-learning and e-assessment? 

Q2. What are the primary forms of e-assessment that you are dealing with? 

Q3. What is you knowledge and contact with tool boxes and/or use of them? 

Part B. Experiences (strengths and shortcomings) around e-assessment 
issues 

Q1 . What comes to mind when we talk about e-assessment and e-assessment tools? 
For example, some people describe e-assessment as: 

• a broadly-based concept that covers a wide range of activities where digital 
technologies are used in assessment, including the designing and delivery of 
assessments, and in marking 

• often discussed are the end-to-end electronic assessment processes where 
ICT is used for the presentation of assessment activity, and the recording of 
responses 
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• methods include self and group assessment, chat and bulletin boards, group 
and collaborative methods, and simulation and project-based assessment; 
also e-portfolios. 

Q2. Overall, do you support the increased use of the online environment to expand 
possibilities around learning and the assessment of learning? Why? Why not? 

Q3. Some people report a perception that there is a general lack of confidence in 
assessment that is e-assessment rather than more traditional forms of assessment. 
Do you share this perception? 

Q4. Which aspects of e-assessment cause you concern? If they do not concern you, 
what do you know others are concerned about? 

Q5. Some people also say there is a growing divide between practitioners and 
auditors around the use of e-assessment? Do you agree or not? If yes, what are the 
main sources for this division? 

Q6.What can be done to manage such issues better as we go forward with AQTF 2007? 

Q7.What do you think is reasonable in terms of how to gain authentication that a 
person has done the on line assessment? 

Q.8. In terms of the nature of VET skills/competencies, which skills/competencies do 
they feel are appropriate to be assessed online and by which approach? Why? 

Q9. What skills-competencies do they feel are not appropriately assessed online and 
what approaches are more appropriate? Why? 

Q1 0. What do you see are the particular strengths of various methods of 
e-assessment that you are primarily managing or you are seeing used? Please name 
them, and then talk this through more. 

Q1 1 . What do you see are the particular shortcomings of methods of e-assessment 
that you are primarily managing or you are seeing used? Please name them, and 
then talk this through more. 

Part C. Solutions and the future 

Q1 . What solutions do you see currently operating or what would you propose in the 
future around managing the challenges better than at present that are associated 
with the growing use of e-assessment? 

Q2. What do you think about solutions that are emerging around the use of the 
learning management systems or other software that supports the resolution of these 
concerns (e.g. around checking who is doing the assessment)? 

Q3. Overall, what are the implications of these and other solutions you have raised 
for 

• the AQTF 2007 guidelines? 

• for practitioners? 

• for auditors? 

Q4 Can you nominate examples of good practice around e-assessment and the 
auditing function of e-assessment and, tell me why you have nominated these 
examples. 

Q5. Can you give me other contacts of people I should talk to further about the 
issues we have examined. 
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For more information: 

Benchmarking and Research business activity 

Annie Fergusson 

Phone: (08) 8348 4071 

Email: Annie.Fergusson@sa.gov.au 

Website: flexiblelearning.net.au/research 

Australian Flexible Learning Framework 

Phone: (07) 3307 4700 

Email: enquiries@flexiblelearning.net.au 

Website: flexiblelearninq.net.au 
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